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DETAILED ACTION 

1. This action is responsive to communications: Response filed on 1/19/05. 

2. Claims 15-17, 22-23, 41-43, and 48-49 are cancelled. Claims 1-14, 18-21, 24-40, 44-47, 
and 50-52 are pending in the case. Claims 1 and 27 are independent claims. 

Claim Rejections - 35 USC § 103 

3. The following is a quotation of 35 U.S. C. 103(a) which forms the basis for all 
obviousness rejections set forth in this Office action: 

(a) A patent may not be obtained though the invention is not identically disclosed or described as set forth in 
section 102 of this title, if the differences between the subject matter sought to be patented and the prior art are 
such that the subject matter as a whole would have been obvious at the time the invention was made to a person 
having ordinary skill in the art to which said subject matter pertains. Patentability shall not be negatived by the 
manner in which the invention was made. 

4. Claims 1-7, 18-19, 27-33, and 44-45 are rejected under 35 U.S.C. 103(a) as being 
unpatentable over U.S. Patent Number 5,708,845 to Wistendahl et al., issued January 13, 
1998, in view of U.S. Patent Number 6,311,178 Bl to Bi et al., issued October 30, 2001, filed 
September 9, 1998 and U.S. Patent Number 4,845,697 to Giddings, issued July 4, 1989, filed 
November 27,1987. 

With respect to the rejection of each dependent claim below, the preceding rejection(s) of 
the relevant base claim(s) is incorporated therein. 

Regarding independent claims 1 and 27, Wistendahl et al. teach (a) in response to a signal of 
interest at a particular time during the temporal document, identifying a portion of the temporal 
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document for which related documents are to be found and (b) selecting text associated with the 
portion of the temporal document identified. (Wistendahl et al., col. 7, lines 55-59.) 

Further, Wistendahl et al. do not teach (c) weighting each term in the selected text by a 
function W(t) according to the time t at which the term occurs relative to the time at which the 
signal of interest occurs and (d) finding the related documents by use of information retrieval 
techniques as applied to the weighted terms. 

However, Bi et al. teach weighting search terms and further teach that this step provides 
the benefit of giving users search results with which they are more likely to be satisfied. (Bi et 
al., col. 2, lines 38-48.) Moreover, Giddings would have suggested to one of ordinary skill in the 
art combining and extending Wistendahl et al. and Bi et al. to use a function W(t) according to 
the time t at which the term occurs relative to the time at which the signal of interest occurs 
inasmuch as Giddings teaches a technique for searching video data in which, upon a failure to 
find a predetermined frame, a search is conducted back and forth of successive frames 
surrounding the given frame for a given time period. (Giddings, col. 5, lines 27-42.) 

One of ordinary skill in the art would have been further motivated to implement a 
function W(t) by the recognition that a search result close in time to the signal of interest would 
have been more likely to have been relevant. Therefore, it would have been obvious to one of 
ordinary skill in the art at the time of the invention to have combined Wistendahl et al., Bi et al., 
and Giddings to weight each term in the selected text by a function W(t) according to the time t 
at which the term occurs relative to the time at which the signal of interest occurs and (d) find the 
related documents by use of information retrieval techniques as applied to the weighted terms. 
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Regarding dependent claims 2 and 28, Wistendahl et al. teach that the temporal document is 
video and audio material inasmuch as they teach a movie with audio and video components. 
(Wistendahl et al., col. 7, lines 55-56.) 

Regarding dependent claims 3 and 29, Wistendahl et al. teach that the video material is stored 
on a video server inasmuch as this element is inherent in the teaching of large digital libraries 
transmitted to subscribers. (Wistendahl et al., col. 6, line, 58, col. 7, line 6.) 
Regarding dependent claims 4 and 30, Wistendahl et al. do not explicitly teach applying 
speech recognition techniques to the audio component of the identified temporal document. 
However, Logan et al. suggest this step inasmuch as they teach searching a collateral text file 
(Logan et al., col. 39, lines 10-14) and also teach voice input to navigate program files. (Logan et 
al., Abstract.) Therefore, it would have been obvious to one of ordinary skill in the art at the time 
of the invention to have applied speech recognition techniques to the audio component of the 
identified temporal document. 

Regarding dependent claims 5 and 31, Wistendahl et al. teach pop-up movie trivia, which is 
the equivalent of close-captioned text. (Wistendahl et al., col. 7, lines 55-59.) 
Regarding dependent claims 6 and 32, Wistendahl et al. teach the temporal document 
including text as discussed above regarding claims 5 and 3 1 . 

Regarding dependent claims 7 and 33, Wistendahl et al. teach that the document text varies 
with time and the selected text is that portion of the temporal document identified. (Wistendahl et 
al., col. 7, lines 53-59.) 

Regarding dependent claims 18 and 44, Wistendahl et al.. teach accessing related documents 
through the Internet. (Wistendahl et al., col. 5, lines 14-15.) 
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Regarding dependent claims 19 and 45, Wistendahl et al. teach selecting the related documents 
from among a collection of documents which may be accessed through the Internet. (Wistendahl 
et al, col. 5, lines 14-15; col. 8, lines 66-67.) Wistendahl et al. do not explicitly teach utilizing 
databases comprising information about the collection but it would have been obvious to one of 
ordinary skill in the art at the time of the invention to utilize such databases because it was well 
known in the art that databases were the most common devices in which collections of 
information were stored and one of ordinary skill in the art would have recognized that utilizing 
a database to select documents was an efficient and reliable way of doing so. 

5. Claims 8-14 and 34-40 are rejected under 35 U.S.C. 103(a) as being unpatentable 
over Wistendahl et al. in view of Bi et al. and Giddings and further in view of U.S. Patent 
Number 6,199,076 B 1 to Logan et al., issued March 6, 2001, filed October 2, 1996. 
Regarding dependent claims 8 and 34, Wistendahl et al. does not teach the document text 
including news bulletins, weather, sports scores, or stock information. However, Logan et al. 
suggest extending Wistendahl et al. to include such a step inasmuch as they teach that there is a 
need to be able to search through news, weather, and business data. (Logan et al., col. 1, line 10 - 
col. 2, line 3.) 

Regarding dependent claims 9 and 35, Wistendahl et al. do not teach that W(t) is equal for all 
times between t, before the signal of interest is given and t2 before the signal of interest and zero 
for all other times. However, Logan et al. teach searching a program segment in which W(t) is 
equal for all times between ti before the signal of interest is given and t2 before the signal of 
interest and zero for all other times. (Logan et al, col. 39, lines 10-19.) Moreover, one of 
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ordinary skill in the art would have recognized that a user might want to search within a given 
time frame only and not without it. Therefore, it would have been obvious to one of ordinary 
skill in the art at the time of the invention to have specified that W(t) is equal for all times 
between ti before the signal of interest is given and tz before the signal of interest and zero for all 
other times. 

Regarding dependent claims 10 and 36, Wistendahl et al. do not explicitly teach that ti is 2 
seconds and t2 is 30 seconds. However, Wistendahl et al. do teach that the concept of setting 
time intervals during video display was known in the art. (Wistendahl et al., col. 5, lines 23-28.) 
Moreover, one of ordinary skill in the art at the time of the invention would have recognized that 
the interval within a search was done should start a few seconds, i.e., 2 seconds before the signal 
of interest was received to account for the user's reaction time and that the search should go a 
certain amount back in time, i. e., 30 seconds. 

Regarding dependent claims 11 and 37, Wistendahl et al. do not teach that W(t) is equal for all 
times between t, before the signal of interest is given and t2 before the signal of interest and 
decreases from t, until the time of the signal, and increases from a time t3 before the signal is 
given to the time t2, and is zero for all other times. Regarding the recitation that W(t) is equal 
for all times between t, before the signal of interest is given and t2 before the signal of interest, 
the rejection of claims 9 and 35 above is fully incorporated herein. 

Further, it would have been obvious to one of ordinary skill in the art at the time of the 
invention to decrease W(t) from t, until the time of the signal because one of ordinary skill in the 
art would have recognized that the closer in time to the signal of interest the more likely it was 
that elapsed time was due only to a user's reaction time and not to interest in the elapsed material. 
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Further, it would have been obvious to one of ordinary skill in the art at the time of the 
invention to decrease W(t) from t2 until t3 because one of ordinary skill in the art would have 
recognized that, up to a certain point, as the time got closer to the time when the signal of interest 
was expressed, it would be more likely that time period contained the subject matter in which the 
user was interested. 

Regarding dependent claims 12 and 38, Wistendahl et al. do not explicitly teach that tl is 2 
seconds and t2 is 15 seconds, and t3 is 30 seconds. However, Wistendahl et al. do teach that the 
concept of setting time intervals during video display was known in the art. (Wistendahl et al., 
col. 5, lines 23-28.) Moreover, one of ordinary skill in the art at the time of the invention would 
have recognized that the interval within a search was done should start a few seconds, i.e., 2 
seconds before the signal of interest was received to account for the user's reaction time and that 
the search should go a certain amount back in time, i. e., 30 seconds, and also that there would be 
a certain point, i. e., 15 seconds, within the 30 seconds at which the likelihood of finding relevant 
subject matter would begin to diminish. 

Regarding dependent claims 13 and 39, Wistendahl et al. does not disclose W(t) behaving 
linearly, but it would have been obvious to one of ordinary skill in the art at the time of the 
invention to have it do so because one of ordinary skill in the art would have recognized that the 
likelihood of finding relevant search results would most likely increase or decrease linearly with 
time. 

Regarding dependent claims 14 and 40, the rejection of claims 12 and 38 above is fully 
incorporated herein. 
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6. Claims 20-21, 24-25, and 46-47, and 50-51 are rejected under 35 U.S.C. 103(a) as 
being unpatentable over Wistendahl et al., Bi et al., and Giddings as applied to claims 19 
and 45 above, and further in view of U.S. Patent Number 6,182,065 Bl to Yeomans, issued 
January 30, 2001, filed April 22, 1998. 

Regarding dependent claims 20 and 46, Wistendahl et al. do not teach selecting related 
documents according to scores achieved according to a formula depending on the occurrence of 
terms which occur in text associated with the portion of the temporal document identified, where 
each term is weighted by a function W(t) according to the time t at which the term occurs relative 
to the time at which the signal of interest occurs. 

However, Yeomans teaches weighting search results according to their predicted 
relevance. (Yeomans, col. 4, lines 60-62.) One of ordinary skill in the art would have recognized 
that weighting allowed users to see more relevant search results, and would also have recognized 
that search results may have been more or less relevant according to where they occurred in time. 
Therefore, it would have been obvious to one of ordinary skill in the art at the time of the 
invention to implement the steps recited in claims 20 and 46. 

Regarding dependent claims 21 and 47, Wistendahl et al. do not teach, but it would have been 
obvious to one of ordinary skill in the art to implement, selecting a predetermined number of 
documents, 1000, because it was well known in the art to limit search results to a predetermined 
number and one of ordinary skill in the art at the time of the invention would have recognized 
that this provided the benefit of not overwhelming the user, and moreover would have 
recognized that 1,000 documents was an upper limit of the number of documents that could 
comfortably be retrieved. 
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Regarding dependent claims 24 and 50, Wistendahl et al. do not teach, but it would have been 
obvious to one of ordinary skill in the art to implement, the step of using terms in portions of the 
document other than the identified portion in calculating scores because one of ordinary skill in 
the art at the time of the invention would have recognized that such terms could have a bearing 
on whether the document was relevant to the user's signal of interest. 

Regarding dependent claims 25 and 51, Wistendahl et al. do not teach, but it would have been 
obvious to one of ordinary skill in the art to implement., the step of having the determination of 
documents receiving the highest scores carried out using compressed document surrogates 
because one of ordinary skill in the art at the time of the invention would have recognized that 
working with compressed document surrogates would have been more efficient than working 
with the full document. 



Allowable Subject Matter 
7. Claims 26 and 52 are objected to as being dependent upon a rejected base claim, but 
would be allowable if rewritten in independent form including all of the limitations of the base 
claim and any intervening claims. 

A statement of reasons for the indication of allowable subject matter was given in the 
previous action mailed on 3/17/03. 
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Response to Arguments 
8. Applicants arguments filed 1/19/05 have been fully considered but they are not 
persuasive. 

Regarding Applicant's remarks on pages 10-11 and 15: 

Referring to claims 1 and 27, the Examiner's understanding of Applicant's argument is 
that Wistendahl does not teach identifying a portion within the temporal document to find related 
documents based on the time at which the portion within the temporal document occurred. In 
other words, the time within the temporal document such as audio or video material can be 
selected or identified by the user (signal of interest) to find related documents associated with 
that time. 

However, the Examiner has interpreted the claim language of limitations (a) and (b) as 
wherein based on the user clicking on a portion within the temporal document (audio or video), 
related documents or text associated with the portion is identified and selected. The user desires 
to click the portion at his/her time to retrieve data or documents related to the selected portion. 

Wistendahl teaches when the user clicks on hotspots or objects within the different scenes 
of a film, data such as trivia games or movie info associated with the hotspot or object is 
displayed to the user in response to the time of clicking (col. 7, lines 55-59). Further, on col. 8, 
lines 55-66 teaches trivia information can be hyperlinked to an object such as Maltese falcon 
within the movie; wherein the user can click on the object to display text or graphics explaining 
the Maltese falcon. 

Regarding Applicant's remarks on page 12, last paragraph, Applicant argues that the 
"text associated with the portion of the temporal document identified" can be closed captions or 
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spoken words from an audio track, however, this feature is not claimed in independent claims 1 
and 27. 

Regarding Applicant's remarks on pages 13-15: 

Applicant argues that Bi and Giddings does not teach weighting each term in the selected 
text by a function W(t) according to the time t at which the term occurs relative to the time at 
which the signal of interest occurs. 

Bi is a computer matching system used by a plurality of users (see Abstract); wherein the 
user enters a search criteria or offers a search requirement (col. 3, lines 54-60, see Figure 1); a 
search is perform in response to the user's search requirement or criteria (signal of interest 
occurs); the system weights the matching results associated with the elements of requirement 
entered by the user (col. 2, lines 38-48). The search engine is responsive to the time the user 
submits the request. 

Giddings teaches a searching mechanism for finding an acceptable frame within a 
prescribed distance from the originally selected frame number desired by the user (signal of 
interest); the search is conducted back and forth of successive frames surrounding the given 
frame for a given time period (on col. 5, lines 27-42). 

Therefore, Bi in combination with Giddings does teach and suggest weighting each term 
in the selected text by a function W(t) according to the time t at which the term occurs relative to 
the time at which the signal of interest occurs. 



Application/Control Number: 09/430,437 Page 12 

Art Unit: 2176 

Conclusion 

9. The prior art made of record and not relied upon is considered pertinent to applicant's 
disclosure. 

Mukhopadhyay et al., "Passive Capture and Structuring of Lectures", 10/1999, ACM 
Multimedia '99, pages 1-11. Mukhopadhyay discloses automatically synchronizing or 
matching textual document with video images. 

10. THIS ACTION IS MADE FINAL. Applicant is reminded of the extension of time 
policy as set forth in 37 CFR 1.136(a). 

A shortened statutory period for reply to this final action is set to expire THREE 
MONTHS from the mailing date of this action. In the event a first reply is filed within TWO 
MONTHS of the mailing date of this final action and the advisory action is not mailed until after 
the end of the THREE-MONTH shortened statutory period, then the shortened statutory period 
will expire on the date the advisory action is mailed, and any extension fee pursuant to 37 
CFR 1.136(a) will be calculated from the mailing date of the advisory action. In no event, 
however, will the statutory period for reply expire later than SIX MONTHS from the mailing 
date of this final action. 

Any inquiry concerning this communication or earlier communications from the 
examiner should be directed to Almari Yuan whose telephone number is 571-272-4104. The 
examiner can normally be reached on Mondays - Fridays (8:30am - 5:00pm). 
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If attempts to reach the examiner by telephone are unsuccessful, the examiner's 
supervisor, Joseph Feild, can be reached on 571-272-4090. The fax phone number for the 
organization where this application or proceeding is assigned is 703-872-9306. 

Information regarding the status of an application may be obtained from the Patent 
Application Information Retrieval (PAIR) system. Status information for published applications 
may be obtained from either Private PAIR or Public PAIR. Status information for unpublished 
applications is available through Private PAIR only. For more information about the PAIR 
system, see http://pair-direct.uspto.gov. Should you have questions on access to the Private PAIR 
system, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). 



AY 

March 31, 2005 



SUPERVISORY PATENT EXAMINER 




